Investigating the Behaviour Of

There is a spectrum of methods for learning robot control. At one end there are model-free methods (eg. Q-learning, AHC, bucket brigade), and at the other there are model-based methods, (eg. dynamic programming by value or policy iteration). The advantage of one technique is the weakness of the other. Model-based methods use experience eeectively, but are computationally expensive; model-free methods are cheap computationally, but require an order of magnitude more experience. In the middle ground there are now methods like Q-DYNA 4] and Prioritised Sweeping 1] which use a learned model to speed temporal credit assignment. The optimal trade-oo is dependent on the relative cost of experience and computation for particular tasks. Unfortunately we frequently do not know the cost balance for a particular task. Hence the ultimate goal of this work is to understand more about the sorts of methods that might work well on a wide variety of cost ratios, and in particular how model free methods might be extended. In this paper we examine the behaviour of one such model-free algorithm, Q() 2]. This algorithm shows promise because it combines the best features of Sutton's TD() algorithm 5] with those of Watkins Q-learning 6]. Despite being a model-free algorithm, it has been reported to outperform Prioritised Sweeping, the current best method for learning a policy and a model at the same time. Here we look at the eeect on its performance of using replacing or accumulating traces, and at the problem of exploration sensitivity. 1 t = 0; ^ Q(x; a) = 0 and T r(x; a) = 0; 8x; a 2 a t = arg max a (^ Q(x t ; a)) 3 e 0 t = r t + ^ V t (x t+1) ? ^ Q t (x t ; a t) e t = r t + ^ V t (x t+1) ? ^ V t (x t) 4 8x; a do T r(x; a) = T r(x; a) ^ Q t+1 (x; a) = ^ Q t (x; a) + T r(x; a)e t 5 ^ Q t+1 (x t ; a t) = ^ Q t (x t ; a t) + e 0 t 6 T r(x t ; a t) = T r(x t ; a t) + 1 7 step 2 x t is the state at time t. a t is the action chosen at time t. …